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20 December 1978 


MEMORANDUM FOR THE RECORD 


2. The purpose of this memorandum is to summarize that discussion. 


25X1 a. [—_Jopened the meeting by explaining how he has been 
using the findings of the recent report on "S&TI Support to Developers 
of Military Tactics and Doctrine" within the Army to tie S&TI produc- 
tion activities to the potential users of the resulting products. Bill 


also provided a status report on py indicated 25x14 
that the final report on the S&TI evaluation process should be avail- 


able by early January 1979. Finally, Bill argued for extending the 
existing contract to investigate S&aTI relationships with the testers 
and evaluators of weapons systems, threat writers and national-level 
consumers of S&TI products. 


25X1 b. [noted that 45 percent of the production budget is 
devoted to the production of S&TI; PAO should not endorse the idea 
that CIA S&TI production activities should be more responsive to DoD 
requirements for S&TI support; the IC Staff is in no position to 
recommend how DoD should go about solving the problems highlighted 
in the latest report; there is concern about how much more analytical 
effort can be profitably invested in trying to understand S&TI produc- 
tion activities. Bill also indicated that S&TI production generally 
produces only intermediary goods--and not products which are of general 
interest at the policymaking level of the government. Finally, Bill 
noted that much of the S&TI production is devoted to producing reports 
which are read only by intelligence personnel--and not by consumers 
outside of the "intelligence fammily." 


25X1 c. L______—|made the following observations: the report on 
Support to developers of military tactics and doctrine makes a number 
of broad assertions which are not generally true of all S&TI users 
within DoD; the study team focused too much attention on the non- 
availability of and/or lack of familiarity with the CAST and not 
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SUBJECT: Presearch S&TI Reports 


enough attention on the informal mechanisms which exist to surface 
intelligence requirements and satisfy consumer needs; PAO will review 

the latest report and let aes when it can be released; Al 25X1 
also told Bill that anything he wants to do with respect to possible 

follow-on efforts should not be premised on any contractual support 

from PAO; PAO is not prepared to extend the existing contract at this 

time, primarily because several projects, which require contractor 

support, have a much higher priority within PAO. 


3. The discussion reached these conclusions: 


25X1 ae [complete its report on the S&TI evaluation 
process by early January. 


b. PAO will review ea and make a 25X14 
determination of when and how they can be distributed. 


25X1 c. PAO will let L_______]know if any follow-on requirements 
develop with respect to his recommendations of 27 October 1978. 
25X1 d.[——sdwill not submit an unsolicited proposal to do any 


follow-on work unless PAO so advisespss—‘“s*CS*s*sSsSsSCSY 25X1 


4. After the Ls ee I the undersigned with a 25X14 
copy of the first portion o @ final report on the evaluation process. 
See attachment. 


25X1 


Attachment: 
Systematic Evaluation of Intelligence 
for Product Improvement and Program 
Justification dtd December 22, 1978 


cc. 
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ABSTRACT 


(U) This report summarizes results of a study of two problems 
faced by the intelligence community: (1) lack of meaningful 
feedback from users on levels of satisfaction so as to cause 
product improvement and (2) nonexistence of qualitative or quan- 
titative data justifying intelligence programs for use in making 
budget decisions, It builds on previous work which identified 
problems in S&TI support to R&D and devised an improved system 
for providing that support. The study demonstrates wider applic- 
ability of the support system and shows that the component for 
evaluation of intelligence products could be used to develop data 
for decisionmaking by intelligence managers, particularly in the 
product improvement, source assessment, and budget processes. 


i 
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I. PREFACE 


BACKGROUND 


1.1 (U) This is the fourth technical report of a research 
effort that was undertaken to improve utilization of scientific 
and technical intelligence (S§&TI) by the R&D community. Prob- 
lems uncovered and solutions developed have been shown to have 
utility beyond the original narrow focus of S§TI in support of 
R&D. Since this report, dealing with systematic evaluation as 

a basic means for intelligence product improvement and program 
justification, logically follows and is based on the three pre- 
vious reports, they are described in the following summary para- 
graphs. 


ie (U) The initial product of this effort was a comprehen- 
sive IC Staff report in October 1977 entitled "Intelligence 
Community Support to Research and Development (U)." That report 
described community deficiencies in S§&TI support to R&D, identi- 
fied problem areas, and gave recommendations for improvement. 


i45 (U) The first study report showed there is very poor 
linkage between S&T production agencies and the large, widely- 
dispersed body of DoD R&D managers. As a result, producers of 
S&TI have only vague, imprecise, largely intuitive knowledge of 
the identity and needs of a key customer group. Tasking of 
S&TI producers is neither comprehensive nor systematic. Defec- 
tive communication between producer and user also means that 
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often the user does not receive the intelligence he needs and, 
conversely, often gets irrelevant or unresponsive products. 


1.4 (U) The initial study report also outlined an uncompli- 
cated system to close the gap between S&TI producer and R&D user. 
The system is applicable to all community members within exist- 
ing organizations and command lines. Among the study recommen- 
dations was that the support system be adopted community-wide. 
All findings and recommendations of the report were briefed ex- 
tensively to upper level management in the intelligence community 
during late 1977 and early 1978. 


18 (U) The initial report pointed out that the disconnect 
between producer and user is a costly failure within the indi- 
vidual military services. The difficulties are magnified when 
potential users are in a service different from a producer. 
The poorest linkage is that between the CIA as a producer of 
S&TI and users in the DoD R&D community. As a result a second 
study addressed this problem and provided recommendations for 
solution. That study, entitled "The Utility and Accessibility 
of CIA S&TI Products in Support of DoD Materiel Acquisition (U)" 
was completed in July 1978. Following a review of the study 
‘report, CIA has acted on certain of the recommendations which 
were within its purview. Other recommendations are being 
staffed. 


1.6 (U) A third report entitled "Intelligence Community 
Support to Developers of Tactics and Doctrine (U)," completed 

in October 1978, demonstrated, among other things, that the sys- 
tem developed for S&TI support to R&D could be used to support 
managers of programs to develop tactics and doctrine. It also 
showed that the system could be used to ensure that the intelli- 
gence needs of any group of users could be met comprehensively 


and economically by producers of any form of intelligence. 


2 
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PROBLEMS ADDRESSED BY STUDY 


La? (U) There are two principal problems addressed in this 
technical report: 


a. The intelligence community does not have a 
system to provide meaningful feedback from 
users regarding levels of satisfaction, so 
as to cause product improvement. Whatever 
evaluation takes place is done by producers 
in the absence of rational and systematic 


inputs from customers. 


b. Intelligence community budget decisions are 
made without qualitative and quantitative 
management information on program justifica- 
tion. Congressional staffs and the Defense 
Audit Service are increasingly aware and 
critical of the traditional practice of 
preparing intelligence programs without doc- 
umentable justification based on needs of 
users. Both know that wastage exists, since 
lack of input by users--including evaluations-- 
must result in production by intuition. 


OBJECTIVES 


1.8 (U) The purposes of the study effort described in this 
technical report were to: 


a. Describe existing practices in the intelli- 
gence community for evaluating intelligence 
products. 


3 
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b. Identify problem areas in the product evalua- 
tion process, 


c. Devise practical solutions to identified eval- 
uation problems that would assist in intelli- 
gence product improvement and budget justifi- 
cation. 


d. Determine if solutions to product evaluation 
problems were applicable to other areas such 
as assessment of collection systems or sources. 


APPROACH 


1.9 (U) During the course of interviews for the three pre- 
vious technical reports described in paragraphs 1.2-1.6, obser- 
vations and data were gathered concerning intelligence product 
evaluation. As a result of briefing the October 1977 report 

at the decisionmaking level in OSD, DIA, CIA, and ICS, oral and 
written comments were received. 


1.10 (U) The study team selectively interviewed at the local 

and intermediate management levels as well as at the Washington 
headquarters level in order to develop useful data. Team mem- 
bers placed emphasis on the needs of users and problems of sup- 
porting them at the local level. Interviewers encouraged sug- 
gestions relating to preparation of documentation, ease or dif- 
ficulty in use of evaluation arrangements, and desirable features 
for a community-wide evaluation system. 


1.11 (U) Analysis in this study effort was aimed at devising 
an evaluation system for intelligence products which would have 
uniform utility for product improvement throughout the intelli- 
gence community. The approach taken was that the evaluation 


4 
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system should be so uncomplicated and direct that it could be 


applied universally. Such an evaluation system could be extended 


down to the level of collection systems and even sources and 
would provide the basis for rational justification of budgets 


at all levels. 


5 
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II. SYSTEMATIC EVALUATION OF INTELLIGENCE PRODUCTS 


NEED FOR EVALUATION SYSTEM 


died (U) A community-wide, systematic evaluation process 

is needed to develop information for managers of intelligence 
production, An evaluation process will provide the basis for 
answering the following questions. 


e Did the product answer the stated require- 
ments of users? 
Is there a continuing need for the product? 
Should the scope of the product be narrowed, 
expanded, or remain the same? 


If the production manager has the answers to these 
questions he can make the near and long-term decisions inherent 
to management control. 


ae (U) Production managers need information derived 

from an evaluation system in order to improve products on a 
rational basis. Concomitantly, the information is needed to 
provide documented substantiation for programs of intelligence. 
Systematic evaluations from users to producers are vital for 

a necessary producer-user dialogue. 


aio (U) No form of intelligence achieves consequence until 
placed in the hands of a user who needs it to perform a mission. 
As a corollary, the significance of intelligence can be deter- 
mined only by the user. 


2.4 (U) For any form of intelligence, the user is the 
technical expert in the field in which he is supported. His 
views and insights on finished intelligence often can lead to 
improvements and to correction of errors. Careful consideration 


Approved For Release 2005/06/09 : CIA-RDP83M00171R000400110005-7 


Approved For Retease 2005/06/09 : CIA-RDP83M0017 14900400110005-7 


(by producers) of the expert opinion of the technically-quali- 
fied user prior to iterations of the intelligence cycle will 
result in responsive products with near-optimum expenditure of 


resources, 


25:5 (U) It follows that without rational evaluation pro- 
cedures that are understood and used throughout the intelli- 
gence community, producers cannot achieve their full potential. 
Moreover, in the absence of well-defined evaluation steps, the 
system to a large extent will drive itself and the intelligence 
community will be perceived as unresponsive to its customers, 


EXISTING ARRANGEMENTS 


2.6 (U) There is now no community-wide system of evaluation 
of intelligence products, nor does any of the services have a 
department-wide system. Within DoD basic guidance for users is 
contained in DIA's"The Intelligence User's Guide (U)", DDM-2600- 
397-78 of January 1978, but no reference is made to evaluations. 


rare | (U) The lack of systematic arrangements in DoD is well 
illustrated by two quotes from the DIA publication "Scientific 
and Techncial Intelligence Production (U)'", DIAM 75-1 of 21 
September 1977. On page 4 is the statement: 


"The S&T intelligence cycle is completed when 
the consumer receives and evaluates the finished 
product in light of his particular needs. If 

it fails to satisfy his needs, it is incumbent 
upon the consumer to provide feedback to the 

S&T production manager and, if appropriate, to 
submit additions, changes, or deletions to the 
standing S&T intelligence requirements in his 
area of interest." 
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"Periodic consumer evaluations of DoD S&T intelligence 
products are authorized and encouraged as a means of insuring 
consumer satisfaction. The Military Departments or designated 
subordinated elements are assigned the responsibility of design- 
ing and conducting surveys and analyzing results. Summaries 
of the results of each survey are to be provided to the DIA 
and the individual responses made available to the DIA task 
monitor for review," 

In the fifteen months since promulgation of DIA on 75-1, 
no results are evident from the foregoing assignment of respon- 
sibility for evaluations of S&T products from DIA to the indiv- 
idual services. There is no rational system to ensure evaluation 
of S&TI or any other form of intelligence. 


2.9 (U) To complete the discussion of existing arrangements 
for evaluation some mention must be made of tearout sheets. 
These are questionnaires disseminated with intelligence products 
that are meant to be filled in by consumers and then returned 

to the producer. Now seldom used, tearout sheets are a tempt- 
ingly simple approach to evaluation. Therefore a discussion of 
them, based on interviews, is informative. 


2.10 (U) The principal objection to the tearout sheet 
approach is that it is uncontrolled. One copy of an intelli- 
gence publication can go to many users, but there is no control 
to ensure that the most qualified or most important recipient 
makes the evaluation. Since the tearout sheets are often exe- 
cuted by unqualified personnel with no strong interest in im- 
proving the publication involved, data deriving from the sheets 
have low validity. 


2.11 (U) In short, production and budgetary decisions 
should not be based on information coming from tearout sheets. 
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Based on the uncontrolled, episodic nature of this evaluation 
technique, it is a finding of this study that the use of tear- 
out sheets is at best useless and a waste of resources. At 
worse tearout sheets are misleading and can be harmful. 


ESSENTIAL FEATURES OF EVALUATION SYSTEM 


ae (U) Figure 2.1, a basic model of the intelligence 
cycle developed for an earlier report uses heavy lines and. 
shading to set out the evaluation phase. It is adapted and 
shown here both to illustrate and emphasize two points that 
are fundamental to the discussion that follows: (a) the intel- 
ligence process must involve the user and (b) the intelligence 
process is an iterative cycle in which all phases, including 
evaluation, are interdependent. 


rape (U) Referring to Figure 2.1, the customer is shown 

in the central oval. It is the customers for whom the whole 
process exists. Working closely with the local intelligence 
officer, the consumer should initiate the cycle by stating his 
needs. A finding of the study team based on interviews at 
several levels of management in the intelligence community 

is that users (managers of R&D and other programs) generally 
do not do a good job of advising intelligence support personnel 
about the current and anticipated status of supported programs. 
As a result, the intelligence community does an inadequate job 
of supporting expensive, important efforts. A system to cor- 
rect this problem has been described in detail in the three 
previous reports of this study series. These reports are sum- 
marized in paragraphs 1.2-1.6 in the Preface of this report. 


2.14 (U) The support system described in the earlier reports 
is designed to meet the intelligence needs of explicitly iden- 
tified users who must complete the cycle by evaluating selected 
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portions of those few products of vital interest to them. It 
is a finding of this study that selective evaluation, rationally 
organized is essential for an easily managed support system. 


oako.. 4B) As shown in Figure 2.1, an evaluation, in effect, 
is a requirement that initiates another iteration of the 
intelligence cycle. The discussion that follows makes the 
intelligence cycle more explicit, particularly the evaluation 
phase. It stresses the importance of systematic customer eval- 
uation in causing the cycle to function efficiently by providing 
management with the information needed for rational decisions 
regarding product improvement and program justification. 


ESSENTIAL FEATURES OF EVALUATION SYSTEM 


2.16 “{U) Table 2.1, Essential Steps for System of Product 
Evaluation (U), shows the actions required for systematic eval- 
uation and the activities involved. It must be stressed at the 
outset that this is neither a complex nor a demanding process. 
If followed, it calls for the management entities shown to carry 
out inherent responsibilities in a logical, consistent way. 


Moreover, since the process is highly selective, involving a 


given user only with discrete portions of highly relevant pub - 
lications, it requires little effort relative to the benefits 


at all levels. 


2.17 (U) If provision were made to perform all of these 
steps in a systematic manner community wide, the Director of 
Central Intelligence could be assured of two important results: 


e That the NFIB budget was being spent to 
support valid needs of bona fide users 


e That documented justification was routinely 
developed for intelligence programs. 
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ESSENTIAL STEPS FOR SYSTEM OF INTELLIGENCE 
PRODUCT EVALUATION (U) 


Define user programs to 


be supported 


Correlate user program 
with program of intelli- 
gence production tasking 


Identify key intelligence 
production tasks from 
user viewpoint 


Isolate intelligence 
gaps 


Submit production 
requirements 


Synthesize and validate 
production requirements 


Program and catalog task- 
ing for discrete products 
Produce publications 


(arrange for collection, 
processing, analysis) 


Disseminate publications 


Evaluate discrete portions 
of selected publications 


Synthesize evaluations 


Forward evaluations to DIA 
and production agency 


Submit new production 
requirements (restart 
cycle) 


Update intelligence 
production tasking 
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Intelligence User 


Local intelligence 
officer/user 


User/local intelligence 
officer 


User/local intelligence 
officer 


Local intelligence 
officer 


Service management control 
DIA/CIA 


Service production 
agencies/CIA/DIA 


DIA/service production 
agencies 


User/local intelligence 
officer 


Service management control 
Service management control 


Local intelligence 
officer/user 


DIA 
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The two foregoing processes do not take place now. And nowhere 
are the necessary steps leading to them more ignored than in 
the phase of evaluation. 


2.18 (U) A brief explanation follows of each of the steps 
shown in Table 2.1. The steps provide a model of the intelli- 
gence process, stressing the evaluation phase. Like any 
model, the steps comprise an abstraction and assist in under- 
standing a seemingly complex, yet straightforward real-world 
process. For ease of reference, Figure 2.2, "Evaluation 
Process For Intelligence Products," has been placed at the end 
of this section as a foldout. The reader may wish to refer to 
it while reading paragraphs 2.19-2.32. 


2.19 (U) Step 1-Define User Programs. Previous studies 
for the IC staff have shown that those programs which should 
receive intelligence support are documented because they have 
gone through an approval-funding process. The initial step in 
the intelligence cycle is for the user, that is the manager of 
an approved, funded effort to state his needs. He should be 
assisted by a local intelligence officer, but only the user can 
say what he needs and does not need. At present, Step 1 is not 
taken on a community-wide basis. 


2.20. (U3 Step 2-Correlate User Program with Program of 
Intelligence Tasking. This is the responsibility of the local 


intelligence officer who supports the manager of funded efforts. 
Previous study reports point out that there are organizations of 
local intelligence officers related to all major programs such as 
R&D, Tactics and Doctrine Development, and the like. At present, 
however, only the Scientific and Technical Intelligence program 
is adequately documented. It is a recommendation of this study 
that tasking for all intelligence programs be documented (see 
Step 7) in the manner of the Catalog of Approved Scientific and 
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Technical Intelligence Tasks (CAST). If this were done, Step Z 
could be easily taken at the local intelligence level for all 
forms of intelligence, not just S§TI. The correlation of the 
user program with intelligence production tasking provides 
management with a range of information and leads logically to 
Steps 3 and 4. It also is essential for Step 9. 


dacbk CU) Step 3-Identify Key Intelligence Production Tasks 
From User Viewpoint. As a result of correlating, the user, 


assisted by the local intelligence officer, specifies those 
portions of planned production (tasking) which are vital to his 
program. In so doing, in a rational system, he assumes an 
obligation for evaluating those portions delimitated. By using 
this highly selective approach, the evaluation process can be 
kept easily manageable. The output of Step 3 is an input to 
Step 10. 


2no2 > (0) Step 4-Isolate Intelligence Gaps. From correlation 
of the user and intelligence programs (Step 2) it is possible to 
isolate intelligence gaps. If the intelligence production task- 
ing will not result in products to satisfy all the needs of a 
given user, the gaps must be made explicit by the user with help 
from the supporting intelligence officer. The results of this 
lead to Step 5. 


2.23 (U) Step 5-Submit Production Requirements. The sequel 


to Step 4 is the preparation and submission of intelligence 
requirements. This is the responsibility of the supporting 
intelligence professional. Step 5 impacts not only on Step 6, 
but is a key input into Step 10, preplanned selective evaluation. 
As was the case with Step 3, the submission of an Intelligence 
Production Requirement (IPR) should call for an evaluation of 
the resultant product. 
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2324 (UY) Step 6-Synthesize and Validate Production Require- 
ments. This obviously needed step should be performed at two 
levels—at the management level for the supporting intelligence 
officers and at the service or departmental management level. 
Using S&TI support to R&D as an example, since it has been 
studied previously, IPRs would go to DARCOM for Army, NAVMAT 
for Navy, and SYSCOM for Air Force. Following consolidation 
they would go to succeeding echelons respectively ACS(I), 
COMNAVINTCOM, and AF/IN. As shown in Figure 2.2, Step 6 leads 
logically into Step Pe 


2425.) Step 7-Program and Catalog Tasking for Discrete 
Products. For DoD this is the responsibility of DIA. It was 


the finding of an earlier study that CIA should also provide 

a catalog of its intelligence production tasking. Figure 2.2 
shows that Step 7 is critical for the entire intelligence 
cycle, including evaluation for it is a necessary condition for 
Step Z and Step 8. 


2.26 ~(U) Step 8-Produce Publications (arrange for collection, 
processing, analysis). This is self-evident and the logical 


result of Step 7. It is the responsibility of the service pro- 
duction agencies, CIA, and DIA. 


Zea7 {U) Step 9-Disseminate Publications. This is a separate, 
important function that will be done imperfectly in the absence 

of information developed in Step 2. If the customer and the 
supporting intelligence officer have properly correlated the 

user and intelligence programs the right publications will get 

to the user for evaluation. Past studies have shown that dis- 
semination is not done well community-wide for DIA or CIA publi- 
cations, yet it is critical for the evaluation process. 
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oeae > CU) Step 10-Evaluate Discrete Portions of Selected 
Publications. It is at this phase that the customer can and 
should have a critical impact on the entire intelligence cycle. 
By giving his reasoned views concerning responsiveness of 
intelligence products, the user-reviewer sets the stage for 
refined requirements and production response. Note that a 
planned, nontaxing evaluation process requires three antecedents- 
accurately disseminated products (Step 9); previously identified 
intelligence tasks that the user considers vital (Step 3); and 
IPRs that the user has initiated (Step 5). Because of data from 
Step 2, the evaluation workload at a given location can be 
easily anticipated. The supporting intelligence officer should 
write the evaluation based on customer views and send it to the 
superior echelon which is responsible for management control 

of support. Note Figure 2.2 shows only the major steps in the 
long-term, steady state functioning of the evaluation process 
within the intelligence cycle. Not shown is an extremely im- 
portant near-term step—arranging for rapid response for easily- 
filled gaps discovered during evaluation, or any other time. 
Long-term, considered responses are important, but the shorter- 
term answers help create the environment of intelligence respon- 
Siveness so important to a healthy relationship between users 
and the intelligence community. 


2.29 (U) Step 11-Synthesize Evaluations. At the management 
echelon above that of local intelligence officers, evaluations 
should be collated, redundancies eleminated and a command 
position established. Review of evaluations at the management 
control level permits an overview of production, leading to 
actions aS important as recommending cancellation of a given 
product. 


2.30 (U) Step 12-Forward Evaluations to DIA and Production 
Agency. As a result of Step 11, evaluations should be forwarded 
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via the command chain to DIA. There is no provision for 
community-wide evaluation of CIA, DIA, or service pro- 
duction agency products. A system to do this is feasible. 
Step 12 is necessary for an orderly update of tasking. 


2.31 (U) Step 13-Submit New Production Requirements. The 
processes of Steps 10 and 11 will disclose intelligence gaps 
requiring long-term action. As a result, users working in 
concert with local intelligence officers should initiate new 
requirements. As noted in paragraph 2.28, shorter-term needs 
are met without resort to IPRs. 


2.52 (DB) Step 14-Update Intelligence Tasking. The input 
from Steps 13 and 14 are critical for making the intelligence 


process responsive to bona fide user needs. Moreover, the 
data regarding current customer requirements would provide 
justification for intelligence programs. This documentable 
justification is at present so incomplete as to be inadequate. 


SUMMARY AND ANTICIPATED RESULTS 


2285: (WU) Product improvement and program justification 
throughout the intelligence community are not possible in the 
absence of a rational evaluation system. An evaluation system 
for general use is easily feasible and would have the following 
major attributes: 


e Economy. The procedures for review should 
not be onerous. They should give a high 
return in improved products for a small 
investment in time from the user and 
intelligence communities. The key to this 
is judicious differentiation between vital 
and generally useful intelligence—from 
the users' viewpoint. 
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Solution 


e Simple community-wide evaluation system 
e Model exists--see Table 2.1 and Figure 2.2 
e Organization exists 


e Documentation exists for supported efforts, 
but some additional intelligence tasking 
needed. 


Recommendations 


e Establish evaluation system for purposes of 
product improvement and program justification 


e Use product evaluation techniques to assess 


collection systems and sources. 
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(U) FIGURE 2.2 
EVALUATION PROCESS FOR INTELLIGENCE PRODUCTS 
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